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Many writers (e.g., Dunnette, 1963; 
Ghiselli & Haire, I960; Guion, 1965; Wal- 
lace, 1965) have e xp ressed concern about 
the difficulties enc ountered in trying to 
predict job performance, and i n establis h^ 
irtgt5ej/a^ In 
ge neral, their misgiving s^-erTter^n' oTind^tj ]^ 
kpw validities ob tai ned and misapplications 
„ of the so-called classic va lidity model." 
To help amelioratej hejse^^ 
jproposed here that the .con cept ofvaiiditv 

and concurrent- situa tions an d introduc e 
jhenorioj ^jffl behav^torarconsistenc^T^y 
consisjtej icx'Qfltid^ 
than that Jkmilj|^^ 

dom, "The ^stj^^^^^^^are^pertor- 
manc e is past performance.^uTprisTngly 
few data sTenrto*e^sTrcTeTtrier support or 
refute this generalization. It deserves con- 
siderably more attention. 



* P. F. Wernimont and J. P. Campbell, "Signs, 
Samples, and Criteria," Journal of Applied Psychology 
52 (1968), 372-76. Copyright 1968 by the American 
Psychological Association. Reprinted/ Adapted by 
permission of the publisher and author. 



SOME HISTORY 

It is perhaps not too difficult to trace the 
steps by which applied psychologists ar- 
rived at their present situation. During 
both World War I and World War II gen- 
eral intelligence and aptitude tests were ef- 
fectively applied to military personnel 
problems. Largely as the result of these 
successes, the techniques developed in 
the armed services were transported to 
the industrial situation and applied to the 
personnel problems of the business organi- 
zation. From a concentration on global 
measures of mental ability, validation ef- 
forts branched out to include measures of 
specific aptitudes, interests, and personality 
dimensions. The process is perhaps most 
clearly illustrated by the efforts of the 
United States Employment Service to vali- 
date the General Aptitude Test Battery 
across a wide range of jobs and occupa- 
tions. In general, testing seemed to be a 
quick, economical, and easy way of obtain- 
ing useful information which removed the 
necessity for putting an individual on the 
job and observing his performance over a 
trial period. 

197 



198 



METHODS OF SELECTION 



It^jgas jn the context of the above effo rts 
-tha t an unfortunate marriage occurred , 
^ namely, the union of the_cjassic validity 
model with the use <£f jests assighg, or indk 
camrs^_of _rjredispositions to b ehave in cer- 
ta in ways (C ro nbach, I960, p. 457),, ra ther 
; ^anarg mplis" of the characteris tjc^egg;-. 
( ior^m^ividuals. A n all too frequent pro- 
cedure was to feed as many signs as possi- 
ble into the classic validity framework in 
hopes that the model itself would somehow 
uncover something useful. T2a£_^rgument 
h ere is that it will be much more fruitful _tet 
focus^n meaningful samples of behavior?" 
rat ^Fthan signs of predispositions,. as _p re- 
4Hg^ o naterperfosmance. 

THE CONSISTENCY MODEL 

To further illustrate the point, consider ^ 
a hypothetical prediction situation in which 
the following five measures are available: 

1. Scores on a mental ability test; 

2. School grade-point average (GPA); 

3. Job-performance criterion at Time 1; 

4. Job-performance criterion at Time 2; 

5. Job-performance criterion at Time 3. 
Obviously, a number of prediction op- 
portunities are possible. Test scores could 
be correlated with GPA; school achieve- 
ment could be correlated with first-year job 
success; or the test scores and GPA could 
be combined in some fashion and the com- 
posite used to predict first-, second-, or 
third-year job performance. All of these 
correlations would be labeled validity coef- 
ficients and all would conform to the classic 
validity model. It is less clear whar lah ^l 
s hould be attached^to the co rrelation he- 

^weejuag L-different measiir?ToTlop"per- 
-ijOfiSHiSHiP^wwoi^d call it validity; many 

(TheJSL^ejeins_toJb^ 

plied p s yc ho lugm fr^o^^held-^h^e^ 
suj££ofessejirially the same behavior, even 



i f they were obtained at two di ffers 
P^jn^J^me- That is, the subtleties of the 
conc^torreliability and the ingredients of 
the classic validity model seem to have in- 
grained the notion that validity is a correla- 
tion between a predictor and a criterion 
and the two should somehow be dissimilar. 

However, each of the 10 correlations 
that one could compute from the above sit- 
uation represents the degree of common 
variation between the two variables, given 
the appropriateness of the linear correla- 
tion model. After all, that is what correla- 
tion is all about. In this sense there is no 
logical reason for saying that some of the 
coefficients represent validity and others 
reliability, although there certainly may be 
in other contexts. 4" implicit or explicit 
i n . s»sff»nre nn the ^p redictor being "differ- 
^aC-Seems^eif^^ jLatbet-^ne 




^ihjLuse in managerial ass essment progra ms. 
Aj^his~poiiiia&~^^ 

the measures to be predir^ 
■^sgj?e measur es of behavio r. For example, 
it JKQuH Be somethjnjfr^^ 
<tQ r - u i e - a kekavio^ 

tionailever gj^^ T or snhiTtq-g^^t^ 
Son? The individual does not alwayTnave 
substantial control over such variables, and, 
even with the more obvious biasing influ- 
ences accounted for, they place a ceiling on 
the maximum predictive efficiency to be 
expected. Furthermore, they are several 
steps removed from actual job behavior. In 
this respect, the_ authors ^are^ very much in 
nette(1966) who argues 
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«fer_accoi^i.r. |nn this ai is the behavior 
retranslation te chnique of Smith linlTKeT^ 
dal (J9m)T~ I he applied' " psyBioTogTst "~ 
should reaffirm his mandate and return to 
the measurement of behavior. Only then 
will one learn by what means, and to what 
extent, an individual has influenced his rate 
of promotion, salary increases, or work 
group's production. 

In general terms, what might the selec- 
tion or prediction procedure look/iike if 

^_Qne tried to apply a consistency model? 

A, First\ a comprehensive study of the job 

STOuldJbeJri^^ of job 

performance^^ 
jrf^Peci^^ 

'"''^vN*^ a thorough search of each appli- 
cants pre vious wor kexp erienceahd educIF 
t ional lustory would be carried out to de- 
termine if any of the relevan t behaviors oT 
outcomes have been required ot him 67 
liave been exhibited in the past. Items and 
"rati ng method s would be developed to fa- 
cilitate judging the frequency of such be- 
haviors, the intensity with which they were 
manifested, the similarity of their context 
to the job situation, and the likelihood that 
they will show up again. These judgments 
can then be related to similar judgments 
concerning significant and consistent as- 
pects of an individual's job behavior. 

Su ch a proc edure places considerable 
emphasis onback^round data and isliimilar 
in torm to the "selection by objective s" 
cxuicepT _of Odiorne and Miller __(1966^ 
However, the aim is to be considerably 
more systematic and to focus on job behav- 
ior and not summary "objectives." 

Afjexjiie-aiaaJysis_ of background da ta it 



mi ght be foun d that the required job he- 
hjtyjors_have n ot been a paFT^ftHe^pli- 
<e ^B^LP£g£iHP£j^^^^§I^^"Id be nec- 
essary to look for the likelihood of that job 
JaeTiavioT^-H-TaT^^ tests'" 
oF^uh Ttlon exercises Anumber of such 
behavior measures are' already being used 
in various management assessment pro- 



Finally , individual pe rforma nce mea- 
su res or psychological varTabTeTlvouid be 
given wider use w hej^ajppropr^fp For ex- 
ample, the Wechsle7~AlIuT^ 
Scale (Wechsler, 1955) might be used to 
assess certain cognitive functions. Notice 
that such a measure is a step closer to actual 
performance sampling than are the usual 
kinds of group intelligence tests. 

How does the above procedure compare 
to conventional practice? The authors hope 
they are not beating at a straw man if the 
usual selection procedure is described as 
follows. First, a thorough job analysis is 
made to discover the types of skills and 
abilities necessary for effective perfor- 
mance. This is similar to the consistency 
approach except that the objective seems 
to be a jump very quickly to a generalized 
statement of skills and abilities rather than 
remaining on the behavioral level. The 
conventional approach next entails a search 
for possible predictors to try out against 
possible criteria. Based on knowledge of 
the personnel selection and individual dif- 
ferences literature, personal experience, 
and "best guesses," some decisions are 
made concerning what predictors to in- 
clude in the initial battery. It is the authors' 
co ntention that th e classic validity model 
has T fnrrpH f" "*^Tn> amfM'tit jrt attention 
on test and inventory measures a t this 
stage^ Witness th eJLarge amount ot "Space 
devoted to a discussion ot 
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the choice seems to be made with little ref- 
erence to the previous job analysis and is 
based on a consideration of "objectivity" 
and relevance to the "ultimate" criterion. 
Unfortunately, even a slight misuse of 
these considerations can lead to criteria 
which are poorly understood. In contrast, 
working within the framework of a consis- 
tency model requires consideration of di- 
mensions of actual job behavior. 

It might be added that the above charac- 
terization of the conventional approach is 
meant to be somewhat idealized. Certain 
departures from the ideal might reinforce 
the use of signs to an even greater extent. 
For example, there is always the clear and 
present danger that the skill requirements 
will be stated in terms of "traits" (e.g., loy- 
alty, resourcefulness, initiative) and thus 
lead even more directly to criteria and pre- 
dictors which are oriented toward underly- 
ing predispositions. 

RELATIONSHIP TO 
OTHER ISSUES 

The consistency notion has direct rele- 
vance for a number of research issues that 
appear frequently in the selection and pre- 
diction literature. Ojje_jmportant implica- 
ti pn is that selection research should focus 
cmJndividuals to a muchjgr eater exten t 
) than Jt has. That is^jherejh ould be mor e 
^emphasis on ln traindiyidual consistency of 

the criterion problem, Ghiselli and Haire 
(I960) point out that intraindividual crite- 
rion performance sometimes varies appre- 
ciably over time, that is, is "dynamic." They 
give two examples of this phenomenon. 
However, after an exhaustive review of the 
literature, Ronan and Prien (1966) con- 
cluded that a general answer to the ques- 
tion, "Is job performance reliable?" is not 
really possible with present data. They go 
on to say that previous research has not 
adequately considered the relevant dimen- 
sions that contribute to job performance 



and very few studies have actually used the 
same criterion measure to assess perfor- 
mance at two or more points in time. In the 
absence of much knowledge concerning 
the stability of relevant job behaviors it 
seems a bit dangerous to apply the classic 
validation model and attempt to generalize 
from a one-time criterion measure to an 
appreciable time span of job behavior. Uti- 
lizing the consistency notion confronts the 
problem directly and forces a consideration 
of what job behaviors are recurring con- 
tributors to effective performance (and 
therefore predictable) and which are not. 

In addition, the adoption_ofsigns as pre- 
d ictors in the context of the classic j nodel 
^ has undoubtedly been a major facto r con- 
t ributing to the lac k of longitudmaTle - 
search . It makes it far too easy to rely on 
concurrent studies, and an enormous 
amount of effort has been expended in that 
direction. E mphasis on behavior sample s 
a nd behavior consistency requires that a 

fQj gBfi r^ ang jgatJ^^ cons ider- 
ation_o£^e ^ of a longitu- 

-dinaLsx.udy. 

The moderato ror subgrouping concept 
also^ seems an integral paTfof the co nsis- 
te ncy ap proach. The basic research aim" is 
^olmd"~su bg rc mps of - people in a particular 
job family lor whom behavior on a partictF" 
Jgj^ performance dimension is consiste nt. 
Subg rouping may be by individual orsitu a- 
tional characteristic s but the necessity is 
dear andhinesc^ aEle. Only within such 
subgroups is lo~ngitudinal prediction pos- 
sible. 

Lastly, the process the authors are advo- 
cating demands a great deal in terms of be- 
ing able to specify the contextual or situa- 
tional factors that influence performance. It 
is extremely important to have some 
knowledge of the stimulus conditions un- 
der which the job behavior is emitted such 
that a more precise comparison to the pre- 
dictor behavior sample can be made. Be- 
cause of present difficulties in specifying 
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the stimulus conditions in an organization 
(e.g., Sells, 1964), this may be the weakest 
link in the entire procedure. However, it is 
also a severe problem for any other predic- 
tion scheme, but is usually not made ex- 
plicit. 

It is important to note that the authors' 
notion of a consistency model does not rest 
on a simple deterministic philosophy and is 
not meant to preclude taking account of 
so-called "emergent" behaviors. Relative to 
"creativity," for example, the question be- 
comes whether or not the indivi dual ha s 

^ve7~exhlb~ited in simil ar contex ts the par- 
ticular kind of creative behavior undercon^ 

^sideration. It a similar context never*ex- 
'~1sT^ rt hc re ac aTch' musTm vestigat e creat ive " 

^performance and^utputs obtained in a jest 

^situation which simulates the contextual 
limitations and requirements in the job sit- 
uation. 

A n additiona l adv antage of the consis- 
tenc vjipproach is tnaTa number of o ldj-ir 
persistent problems fortunately appear to 
flissipate, or at least become signi ficantly 
diminishedrT!o^s7der"the followingT~~~~™^' 
T U taking and response sets — Since 
tqe'emphasis would be on behavior sam- 
ples and not on self-reports of attitudes, 
beliefs, and interests, these kinds of re- 
sponse bias would seem to be less of a 
^pcoblem. 

( 2J Discrimination in testing — Accord- 
ingao Doppelt and Bennett (1967) two 
general charges are often leveled at tests as 
being discriminatory devices: 

^rfH-ack of relevance — It is charged that 
testisems are often not related to the work 
required on the job for which the applicant 
is being considered, and that even where 
relationships can be shown between test 
scores and job success there is no need to 
eliminate low-scoring disadvantaged peo- 
ple since they can be taught the necessary 
skills and knowledge in a training period 
aftef^hiring. 

r (£) Ijnfairness of content — It is further 
mairitained that most existing tests,, espe- 



cially verbal measures, emphasize middle- 
class concepts and information and are, 
therefore, unfair to those who have not 
been exposed to middle-class cultural and 
educational influences. Consequently, the 
low test scores which are earned are not 
indicative of the "true" abilities of the dis- 
advantaged. Predictions of job success 
made from such scores are therefore held 
to be inaccurate. 

The examination of past behaviors simi- 
lar in nature to desired future behavior, 
along with their contextual ramifications, 
plus the added techniques of work samples 
and simulation devices encompassing de- 
sired future behavior, should markedly re- 
duce both the real and imagined severity of 
problems of unfairness in prediction. 

Invasion of privacy — The very na- 
rur£_^of the consistency approach would 
seem to almost entirely eliminate this prob- 
lem. The link between the preemployment 
or prepromotion behavior and job behav- 
ior is direct and obvious for all to see. 

CONCLUDING COMMENTS 

The preceding discussion is meant to be 
critical of the concepts of predictive and 
concurrent validity. Nothing that has been 
said here should be construed as an attack 
on construct validity, although Campbell 
(I960) has pointed out that reliability and 
validity are also frequently confused within 
this concept. Neither do the authors mean 
to give the impression that a full-scale ap- 
plication of the consistency model would 
be without difficulty. Using available crite- 
ria and signs of assumed underlying deter- 
minants within the framework of the classic 
model is certainly easier; however, for 
long-term gains and the eventual under- 
standing of job performance, focusing on 
the measurement of behavior would almost 
certainly pay a higher return on invest- 
ment. 

Some time ago, Goodenough (1949) di- 
chotomized this distinction by referring to 
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signs versus samples as indicators of future 
behavior. Between Hull's (1928) early 
statement of test validities and Ghiselli's 
(1966) more recent review, almost all re- 
search and development efforts have been 
directed at signs. Relatively small benefits 
seem to have resulted. In contrast, some 
recent research efforts directed at samples 
seem to hold out more promise. The 
AT&T studies, which used ratings of be- 
havior in simulated exercises (Bray & 
Grant, 1966), and -the In-basket studies re- 
ported by Lopez (1965) are successful ex- 
amples of employing behavior samples 
with management and administrative per- 
sonnel. Frederiksen (1966) has reported 
considerable data contributing to the con- 
struct validity of the In-basket. In addition, 
Ghiselli (1966) has demonstrated that an 
interview rating based on discussion of spe- 
cific aspects of an individual's previous 
work and educational history had reasona- 
bly high validity, even under very unfavor- 
able circumstances. In a nonbusiness set- 
ting, Gordon (1967) found that a work 
sample yielded relatively high validities for 
predicting final selection into the Peace 
Corps and seemed to be largely indepen- 
dent of the tests that were also included as 
predictors. 

Hopefully, these first few attempts are 
the beginning of a whole new technology 
of behavior sampling and measurement, in 
both real and simulated situations. If this 
technology can be realized and the consis- 
tencies of various relevant behavior dimen- 
sions mapped out, the selection literature 
can cease being apologetic and the predic- 
tion of performance will have begun to be 
understood. 



REFERENCES 

Bray, D. W., & Grant, D. L. The assessment 
center in the measurement of potential for 
business management. Psychological Mono- 
graphs, 1966, 80(17, Whole No. 625). 



Campbell, D. T. Recommendations for APA 
test standards regarding construct, trait, 
and discriminant validity. American Psychol- 
ogist, I960, 15, 546-553. 

Cronbach, L. J. Essentials of pyschological testing. 
(2nd ed.) New York: Harper & Row, 
1960. 

Doppelt, J. P., & Bennett, G. K. Testing job 
applicants from disadvantaged groups. Test 
Service Bulletin (No. 57). New York: Psy- 
chological Corporation, 1967, pp. 1-5. 
Dunnette, M. D. A modified model for test vali- 
dation and research. Journal of Applied Psy- 
chology, 1963, 47, 317-323. 
Dunnette, M. D. Personnel selection and place- 
ment. Belmont, Calif.: Wadsworth, 1966. 
Frederiksen, N. Validation of a simulation tech- 
nique. Organizational Behavior and Human 
Performance, 1966, 1, 87-109. 
Ghiselli, E. E. The validity of occupational apti- 
tude tests. New York: Wiley, 1966. 
Ghiselli, E. E., & Haire, M. The validation of 
- selection tests in the light of the dynamic 
character of criteria. Personnel Psychology, 
1960, 13, 225-231. ' 
Goodenough, F. Mental testing: Its history, prin- 
ciples, and applications. New York: Holt, 
Rinehart & Winston, 1949. 
Gordon, L. V. Clinical, psychometric, and work 
sample approaches in the prediction of suc- 
cess in Peace Corps training. Journal of Ap- 
plied Psychology, 1967,51, 111-119. 
Guion, R. M. Synthetic validity in a small com- 
pany: A demonstration: Personnel Psychol- 
ogy, 1965, 18, 49-65. 
Hull, C. L. Aptitude testing. New York: Har- 

court, Brace Janovich, 1928. 
Lopez, F. M., Jr. Evaluating executive decision 
making: The In-basket technique. New York: 
American Management Association 
1965. 

Odiorne, G. S., & Miller, E. L. Selection by 
objectives: A new approach to managerial 
selection. Management of Personnel Quar- 
terly, 1966, 5(5), 2-10. 

Ronan, W. W., & Prien, E. P. Toward a criterion 
theory: A review and analysis of research and 
opinion. Greensboro, N.C.: Richardson 
Foundation, 1966. 



READING 15 



Sells, S. B. Toward a taxonomy of organizations. 
In W. W. Cooper, H. J. Leavitt, & W. W. 
Shelly, II (Eds.), New perspectives in organi- 
zation research. New York: Wiley, 1964. 

Smith, P. C, & Kendall, L. M. Retranslation of 
expectations: An approach to the construc- 
tion of unambiguous anchors for rating 



scales. Journal of Applied Psychology, 1963 
47,149-155. 

Wallace, S. R. Criteria for what? American Psy- 
chologist, 1965, 20, 411-417. 

Wechsler, D. Manual for the Wechsler Adult In- 
telligence Scale. New York: Psychological 
Corporation, 1955. 



Reading 15 

Content-Oriented Personnel Selection in a Small 
Business Setting* 

DAVID D. ROBINSON 



A "new emphasis" in the prediction of 
job behavior was proposed by Wernimont 
and Campbell (1969). The essence of their 
idea was that the classic model of criterion- 
related validity ought to be replaced by a 
" behavioral consistency" approach to pre- 
diction. Tfi HIapproacn would rpl y _u rj^n^ 
"establishment of consis tencies between 
relev-aiu dimensions of lo^ehlvjor and" 
preemployment-behavior sarng h£r~'nKr 
fained from real or simulated lrt^ aons7 r " 
Guion (1974) pointed out that industrial 
psychologists paid little attention to con- 
tent validity until the term was thrust upon 
them by federal regulations, and concluded 
that content-referenced measurement con- 
stituted a "new window" to be opened. 
Concepts of job-relatedness and due pro- 
fessional care, emphasized by the courts, 
e.g., in Griggs v. Duke Power 1 and Albemarle 
v. Moody 2 have stimulated interest in con- 
tent-oriented methodologies. 



* D. D. Robinson, "Content-Oriented Personnel Se- 
lection in a Small Business Setting," Personnel Psychol- 
ogy 34 (1981), pp. 77-87. Copyright 1981 by Person- 
nel Psychology, Inc. Reprinted/Adapted by permis- 
sion of the publisher and author. 



However, according to Lawshe (1957) 
the newness of the field and the proprietary 
nature of the work done by professionals 
practicing in industry has resulted in a pau- 
city of literature on content validity in em- 
ployment testing except with regard to the 
public sector. Prien (1977) has complained 
that textbooks treat job analysis in such a 
manner as to "suggest that any fool can do 
it," and that by doing so relegate it "to the 
lowest level technician." He asserted the 
job analysis in test selection and criterion 
development must not be done by rum- 
maging around in an organization, but 
through application of highly systematic 
and precise methods. This paper is offered 
in response to the apparent needs identi- 
fied by Lawshe and Prien. Its purpo se is to 
describe a system atic procedure foTT3eTrtP^ 
'lying job contehTanUWiuustrate its applF" 
'^a^ ^ o ^eG ev^^ ^ ^ & el S ca& a i n - a -rtrriafi 
business settingr — — < — • — 



1 Griggs v. Duke Power Co., 401 U.S. 424 (1971). 

2 Albemarle Paper Co. v. Moody, 422 U.S. 407 
(1975). 



